Providing social impact information associated with identified products or businesses

ABSTRACT

Embodiments of the invention are directed to methods and apparatuses for capturing a real-time video stream using a mobile computing device, determining, using a computing device processor, which images from the real-time video stream are associated with goods and businesses consistent with a predetermined social impact, and presenting on a display of the real-time video stream, one or more indicators, each indicator being associated with an image determined to be consistent with a predetermined social impact.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to U.S. Provisional Patent ApplicationSer. No. 61/450,213, filed Mar. 8, 2011, entitled “Real-Time Video ImageAnalysis Applications for Commerce Activity,” and U.S. ProvisionalPatent Application Ser. No. 61/478,403, filed Apr. 22, 2011, entitled“Providing Social Impact Information Associated with Identified Productsor Businesses,” the entirety of each of which is incorporated herein byreference.

BACKGROUND

Modern handheld mobile devices, such as smart phones and the like,combine multiple technologies to provide the user with a vast array ofcapabilities. For example, many smart phones are equipped withsignificant processing power, sophisticated multi-tasking operatingsystems, and high-bandwidth Internet connection capabilities. Moreover,such devices often have additional features that are becomingincreasingly more common as standardized features. Such featuresinclude, but are not limited to, location-determining devices, such asGlobal Positioning System (GPS) devices; sensor devices, such asaccelerometers; and high-resolution video cameras.

As the hardware capabilities of such mobile devices have increased, soto have the applications (i.e., software) that rely on the hardwareadvances. One such example of innovative software is a category known asaugmented reality (AR), or more generally referred to as mediatedreality. One such example of an AR application platform is Layar,available from Layar, Amsterdam, the Netherlands.

The Layar platform technology analyzes location data, compass directiondata, and the like in combination with information related to theobjects, locations or the like in the video stream to create browse-able“hot-spots” or “tags” that are superimposed on the mobile devicedisplay, resulting in an experience described as “reality browsing”.

Consumers are also increasingly interested in the social impact of theproducts they purchase and the businesses they frequent. For example,many consumers are interested in the environmental impact created by themanufacture and sale of the products and services they purchase, someconsumers are interested in purchasing goods and services producedlocally or from a specific geographic location, others desire topurchase goods that have not been manufactured with child labor or thatincorporate resources from a specific geographic location. It can bedifficult for consumers to identify the social impact of specificproducts and businesses at the point where (and when) most purchasingdecisions are made, such as when comparing products in a store orwalking down a street and trying to determine which service provider'sstore to enter.

Therefore, a need exists to implement real-time video analysis, i.e., ARor the like, to assist the user of mobile devices with providing thesocial impact of specific goods and services.

SUMMARY

The following presents a simplified summary of one or more embodimentsof the present invention in order to provide a basic understanding ofsuch embodiments. This summary is not an extensive overview of allcontemplated embodiments, and is intended to neither identify key orcritical elements of all embodiments nor delineate the scope of any orall embodiments. Its sole purpose is to present some concepts of one ormore embodiments in a simplified form as a prelude to the more detaileddescription that is presented later.

Methods, apparatus systems and computer program products are describedherein that provide for using real-time video analysis, such as AR orthe like to assist the user of mobile devices with identifying productsand businesses that are consistent with the user's social values.Through the use of real-time image object recognition, objects, logos,artwork, products, text, locations and other features that can berecognized in a real-time video stream can be matched to data associatedwith such objects to assist the user with identifying the social impactof specific products and businesses. In specific embodiments, the datathat is used to match to the images in the real-time video stream isspecific to financial institutions, such as customer financial behaviorhistory, customer purchase power/transaction history and the like. Inthis regard, some of the embodiments herein disclosed utilize financialinstitution data, which is uniquely specific to financial institutions,in providing information to mobile devices users in connection withreal-time video stream analysis. These embodiments are exemplary.

To the accomplishment of the foregoing and related ends, the one or moreembodiments comprise the features hereinafter fully described andparticularly pointed out in the claims. The following description andthe annexed drawings set forth in detail certain illustrative featuresof the one or more embodiments. These features are indicative, however,of but a few of the various ways in which the principles of variousembodiments may be employed, and this description is intended to includeall such embodiments and their equivalents.

Some embodiments of the present invention provide a method wherein oneor more goods or service providers proximate in location to a mobiledevice are identified by a processor (e.g. the processor of the mobiledevice). It is then determined which goods or service providersproximate in location to the mobile device are consistent with apredetermined social impact. The mobile device then presents on adisplay the one or more indicators associated with a good or serviceprovider determined to be consistent with the predetermined socialimpact profile. In certain embodiments, identifying one or more goods orservice providers involves capturing, via the mobile device, a real-timevideo stream, wherein the real-time video stream includes at least oneimage of a good or service provider. In such embodiments, the processormay implement object recognition processing to identify the goods orservice providers within the real-time video stream.

In other embodiments, identifying one or more goods or service providersinvolves capturing a code located on, or proximate to, one or moreproducts or service providers, wherein the code identifies the productor service provider. In yet other embodiments, the one or more goods orservice providers consistent with a predetermined social impact areidentified by receiving, via the mobile device, wireless communicationfrom a product identifying tag or service provider tag associated withthe goods or service provider. Consistent with certain embodiments,identifying one or more goods or service providers further comprisesdetermining the location of the mobile device and determining via theprocessor, the one or more goods or service providers based on thedetermined location.

In certain embodiments the mobile device is a mobile phone. In someembodiments the real-time video stream is captured by the same mobiledevice that presents the one or more indicators.

In some embodiments of the invention, determining which goods or serviceproviders proximate in location to the mobile device are consistent witha predetermined social impact involves the steps of using a device tocollect data concerning the social values of importance to the user(e.g., current or aggregate of data of importance to the user) and usinga processor to generate a social impact profile for the user based onthe collected data. In some embodiments, the processor then identifiesthe goods and service providers captured in a real-time video stream,correlates those goods and service providers to information regardingthe social impact of the goods and service providers and compares theinformation regarding the social impact of the goods and serviceproviders to the social impact profile of the user.

Consistent with certain embodiments of the invention, the method forcollecting data concerning the social values of greatest importance tothe user can involve using a computer or mobile device for collectinguser defined data concerning the social values of importance to theuser. The user defined data may include, among other things, responsesto survey questions and search terms provided by the user forcorrelation to the goods and service providers identified in thereal-time video stream.

In some embodiments, the method for collecting data concerning thesocial values of greatest importance to the user comprises utilizing aprocessor to analyze the purchasing behavior of the user and identifythe social values of importance to the user from the user's purchasingbehavior. Such purchasing behavior may comprise financial recordsavailable to a financial institution.

In some embodiments of the invention, the method for collecting dataconcerning the social values of greatest importance to the user caninvolve collecting personal data associated with the user, via aprocessor, and identifying the social values of importance to the userfrom the collected personal data. Such personal data may comprise dataavailable to a financial institution, such as data available infinancial records held by a financial institution.

In certain embodiments, presenting one or more indicators on the displayof a mobile device, wherein each indicator is associated with a good orservice provider proximate in location to the mobile device determinedto be consistent with the predetermined social impact comprisessuperimposing the indicator over the real-time video stream presented onthe display. In some embodiments the indicators are selectable by theuser. Selecting the indicators may in some embodiments present the userwith additional information about the goods or service provider orpresent the user with an option to purchase, via the mobile device, oneor more of the goods or a service provided by the service provider.

Embodiments of the invention also provide an apparatus where theapparatus has a device for capturing a real-time video stream, aprocessor, operably connected to the device and configured to determinewhich images from the real-time video stream are associated with goodsor service providers consistent with a predetermined social impactprofile and a display operably connected to the processor for presentingthe real-time video stream with one or more indicators, each indicatorbeing associated with an image determined to be consistent with thepredetermined social impact profile.

In some embodiments of the invention the processor is further configuredto collect data concerning the social values of importance to the user;that data is collected in some embodiments through user defined data, insome embodiments by analyzing the purchasing behavior of the user and insome embodiments by collecting personal data associated with the user.The processor is also configured to use the collected data to generate asocial impact profile for the user, identify goods and service providerscaptured in the real-time video stream, correlate the goods and serviceproviders to information regarding the social impact of each good andbusiness and assign the goods and businesses a social impact profile.Then the processor correlates the social impact provider for the user tothe social impact profile of the goods or service provider. In certainembodiments the social impact of the goods or service providers compriseone or more of an environmental impact, nature of production, productattributes, business attributes or charitable activities. The userdefined data in certain embodiments will be responses to surveyquestions or search terms inputted by the user along with the user'sdesired correlation between the inputted search terms and the goods orservice providers. The apparatus may collect data concerning the socialvalues of importance to the user through purchasing behavior, whereinsuch data comprises the financial records available to a financialinstitution. In certain embodiments, the apparatus may collect dataconcerning the social values of importance to the user by collectingpersonal data associated with the user, such as data available to afinancial institution.

In certain embodiments of the apparatus, the one or more indicatorspresented on the display are superimposed over the real-time videostream presented on the display. In some embodiments the indicators areselectable by the user and may present the user with additionalinformation about the goods or service provider or present the user withan option to purchase, via the device, one or more of the goods or aservice provided by the service provider.

Embodiments of the invention also provide a computer program productcomprising a non-transitory computer-readable medium havingcomputer-executable code stored thereon. In one embodiment, thecomputer-executable code includes: a first code portion configured tocapture, via a mobile device, a real-time video stream, a second codeportion configured to determine which images from the real-time videostream are associated with goods or service providers consistent with apredetermined social impact profile and a third code portion configuredto present on a display the real-time video stream with one or moreindicators, each indicator associated with an image determined to beconsistent with the predetermined social impact profile.

In certain embodiments the second code portion of the computer programproduct is further configured to collect data concerning the socialvalues of importance to the user, generate a social impact profile forthe user and identify the goods and service providers captured in thereal-time video stream. The second code portion may then correlate thegoods or service providers identified in the video stream to informationregarding the social impact of the goods and service providers andcompare the information regarding the social impact of the goods andservice providers to the social impact profile of the user.

The features, functions, and advantages that have been discussed may beachieved independently in various embodiments of the present inventionor may be combined with yet other embodiments, further details of whichcan be seen with reference to the following description and drawings

BRIEF DESCRIPTION OF THE DRAWINGS

Having thus described embodiments of the invention in general terms,reference will now be made to the accompanying drawings, which are notnecessarily drawn to scale, and wherein:

FIG. 1 is a block diagram illustrating a mobile device, in accordancewith an embodiment of the invention;

FIG. 2 is a block diagram illustrating an object recognitionenvironment, in accordance with an embodiment of the invention;

FIG. 3 is a block diagram illustrating a mobile device, in accordancewith an embodiment of the invention;

FIG. 4 is a flow diagram illustrating a process flow for an apparatusfor providing social impact information in association with identifiedproducts or businesses in accordance with an embodiment of theinvention;

FIG. 5 is a flow diagram illustrating a process flow for an apparatusfor determining which images from a real-time video stream areassociated with goods or services consistent with a predetermined socialimpact in accordance with an embodiment of the invention;

FIG. 6 is a flow diagram illustrating a process flow for an apparatusfor collecting data concerning the social values of greatest importanceto a user in accordance with an embodiment of the invention;

FIG. 7 illustrates an exemplary display page of a user interface forcollecting data regarding the social values of importance to the user,in accordance with an embodiment of the invention;

FIG. 8 illustrates an exemplary display page of a user interface forcollecting data regarding the social values of importance to the user,in accordance with an embodiment of the invention;

FIG. 9 illustrates an exemplary display page of a user interface forpurchasing products or services, in accordance with an embodiment of theinvention;

DETAILED DESCRIPTION OF EMBODIMENTS OF THE INVENTION

Embodiments of the present invention will now be described more fullyhereinafter with reference to the accompanying drawings, in which some,but not all, embodiments of the invention are shown. Indeed, theinvention may be embodied in many different forms and should not beconstrued as limited to the embodiments set forth herein; rather, theseembodiments are provided so that this disclosure will satisfy applicablelegal requirements. In the following description, for purposes ofexplanation, numerous specific details are set forth in order to providea thorough understanding of one or more embodiments. It may be evident;however, that such embodiment(s) may be practiced without these specificdetails. Like numbers refer to like elements throughout.

Various embodiments or features will be presented in terms of systemsthat may include a number of devices, components, modules, and the like.It is to be understood and appreciated that the various systems mayinclude additional devices, components, modules, etc. and/or may notinclude all of the devices, components, modules etc. discussed inconnection with the figures. A combination of these approaches may alsobe used.

The steps and/or actions of a method or algorithm described inconnection with the embodiments disclosed herein may be embodieddirectly in hardware, in a software module executed by a processor, orin a combination of the two. A software module may reside in RAM memory,flash memory, ROM memory, EPROM memory, EEPROM memory, registers, a harddisk, a removable disk, a CD-ROM, or any other form of storage mediumknown in the art. An exemplary storage medium may be coupled to theprocessor, such that the processor can read information from, and writeinformation to, the storage medium. In the alternative, the storagemedium may be integral to the processor. Further, in some embodiments,the processor and the storage medium may reside in an ApplicationSpecific Integrated Circuit (ASIC). In the alternative, the processorand the storage medium may reside as discrete components in a computingdevice. Additionally, in some embodiments, the events and/or actions ofa method or algorithm may reside as one or any combination or set ofcodes and/or instructions on a machine-readable medium and/orcomputer-readable medium, which may be incorporated into a computerprogram product.

In one or more embodiments, the functions described may be implementedin hardware, software, firmware, or any combination thereof. Ifimplemented in software, the functions may be stored or transmitted asone or more instructions or code on a computer-readable medium.Computer-readable media includes both computer storage media andcommunication media including any medium that facilitates transfer of acomputer program from one place to another. A storage medium may be anyavailable media that can be accessed by a computer. By way of example,and not limitation, such computer-readable media can comprise RAM, ROM,EEPROM, CD-ROM or other optical disk storage, magnetic disk storage orother magnetic storage devices, or any other medium that can be used tocarry or store desired program code in the form of instructions or datastructures, and that can be accessed by a computer. Also, any connectionmay be termed a computer-readable medium. For example, if software istransmitted from a website, server, or other remote source using acoaxial cable, fiber optic cable, twisted pair, digital subscriber line(DSL), or wireless technologies such as infrared, radio, and microwave,then the coaxial cable, fiber optic cable, twisted pair, DSL, orwireless technologies such as infrared, radio, and microwave areincluded in the definition of medium. “Disk” and “disc”, as used herein,include compact disc (CD), laser disc, optical disc, digital versatiledisc (DVD), floppy disk and blu-ray disc where disks usually reproducedata magnetically, while discs usually reproduce data optically withlasers. Combinations of the above should also be included within thescope of computer-readable media.

Thus, methods, systems, computer programs and the like are hereindisclosed that provide for using real-time video analysis, such as AR orthe like to assist the user of mobile devices with identifying socialimpact information of certain products and businesses. It will beunderstood that “social impact,” as used herein, generally refers to howa product or business affects a given community or society as a whole. Aproduct or businesses' social impact can include, but is not limited to,its environmental impact, including, carbon footprint (i.e. the amountof carbon dioxide emitted due to the consumption of fossil fuels), useof renewable resources, reliance on fossil fuels, associated waste orpollution etc.; the nature of production, including the country oforigin, types of workers, compliance with labor laws, history ofemployment abuses etc.; product attributes, including, whether theproduct is organic or incorporates known carcinogens, etc.; businessattributes, including, ownership of the business, such as whether thebusiness is locally owned, minority owned, publicly traded etc. and thecharitable activities of a business, such as involvement in thecommunity, support of specific charities, corporate donations etc.; andthe like. A “social impact profile” as used herein refers to acategorization of a user, product or business to a grouping orclassification that is correlated to data related to a user's socialvalues or the social impact of a given product or business. Such acategorization can take the form of a numerical ranking, indexing,grading, grouping, typology or other organization that enables anobjective comparison or correlation of social impact information.

The methods, systems, computers programs and the like disclosed hereinuse real-time vision object recognition, objects, logos, artwork,products, text, locations and other features that can be recognized inthe real-time video stream and can be matched to data associated withsuch to assist the user with identifying the social impact of specificproducts and businesses. In specific embodiments, the data that is usedto match the images in the real-time video stream is specific tofinancial institutions, such as customer financial behavior history,customer purchase power/transaction history and the like. In thisregard, many of the embodiments herein disclosed leverage financialinstitution data, which is uniquely specific to financial institution,in providing information to mobile devices users in connection withreal-time video stream analysis.

While embodiments discussed herein are generally described with respectto “real-time video streams” or “real-time video” it will be appreciatedthat the video stream may be captured and stored for later viewing andanalysis. Indeed, in some embodiments video is recorded and stored on amobile device and portions or the entirety of the video may be analyzedat a later time. The later analysis may be conducted on the mobiledevice or loaded onto a different device for analysis. The portions ofthe video that may be stored and analyzed may range from a single frameof video (e.g., a screenshot) to the entirety of the video.Additionally, rather than video, the user may opt to take a stillpicture of the environment to be analyzed immediately or at a latertime. Embodiments in which real-time video, recorded video or stillpictures are analyzed are contemplated herein.

FIG. 4 illustrates a general process flow 400 for providing socialimpact information in association with identified products or businessesin accordance with an embodiment of the invention. As represented byblock 410 a mobile device 100 (FIG. 1) is used to capture a real-timevideo stream. Then, as represented by block 420 a processor 110 (FIG. 1)determines which images from the real-time video stream are associatedwith goods or businesses consistent with a predetermined social impactprofile. As represented by block 420, the real-time video stream is thenpresented on a display 130 (FIG. 3) with one or more indicators, eachindicator 300 being associated with an image determined to be consistentwith a predetermined social impact profile. Embodiments of the process400, and systems and apparatus for performing the process 400, aredescribed in greater detail below with reference to FIGS. 1-9.

FIG. 1 illustrates an embodiment of a mobile device 100 that may beconfigured to execute object recognition and AR functionality, inaccordance with specific embodiments of the present invention. A “mobiledevice” 100 may be any mobile communication device, such as a cellulartelecommunications device (i.e., a cell phone or mobile phone), personaldigital assistant (PDA), smartphone, a mobile Internet accessing device,or other mobile device including, but not limited to portable digitalassistants (PDAs), pagers, mobile televisions, gaming devices, laptopcomputers, tablet computers, cameras, video recorders, audio/videoplayers, radios, GPS devices, and any combination of the aforementioned,wearable computer or the like.

The mobile device 100 may generally include a processor 110 communicablycoupled to such components as a memory 120, user output devices 136,user input devices 140, a network interface 160, a power source 115, aclock or other timer 150, a camera 170, a positioning system device 175,one or more integrated circuits 180, etc.

In some embodiments, the mobile device and/or the server access one ormore databases or datastores (not shown in FIG. 1) to search for and/orretrieve information related to an object and/or marker as discussedherein. In some embodiments, the mobile device and/or the server accessone or more datastores local to the mobile device and/or server and inother embodiments, the mobile device and/or server access datastoresremote to the mobile device and/or server. In some embodiments, themobile device and/or server access both a memory and/or datastore localto the mobile device and/or server as well as a datastore remote fromthe mobile device and/or server

The processor 110, and other processors described herein, may generallyinclude circuitry for implementing communication and/or logic functionsof the mobile device 100. For example, the processor 110 may include adigital signal processor device, a microprocessor device, and variousanalog to digital converters, digital to analog converters, and/or othersupport circuits. Control and signal processing functions of the mobiledevice 100 may be allocated between these devices according to theirrespective capabilities. The processor 110 thus may also include thefunctionality to encode and interleave messages and data prior tomodulation and transmission. The processor 110 may additionally includean internal data modem. Further, the processor 110 may includefunctionality to operate one or more software programs or applications,which may be stored in the memory 120. For example, the processor 110may be capable of operating a connectivity program, such as a webbrowser application 122. The web browser application 122 may then allowthe mobile device 100 to transmit and receive web content, such as, forexample, location-based content and/or other web page content, accordingto a Wireless Application Protocol (WAP), Hypertext Transfer Protocol(HTTP), and/or the like.

The processor 110 may also be capable of operating applications, such asan object recognition application 125. The object recognitionapplication 125 may be downloaded from a server and stored in the memory120 of the mobile device 100. Alternatively, the object recognitionapplication 125 may be pre-installed and stored in a memory in theintegrated circuit 180. In such an embodiment, the user may not need todownload the object recognition application 125 from a server. In someembodiments, the processor 110 may also be capable of operating one ormore applications functioning as an artificial intelligence (“AI”)engine. The processor 110 may recognize objects that it has identifiedin prior uses by way of the AI engine. In this way, the processor 110may recognize specific objects and/or classes of objects, and storeinformation related to the recognized objects in one or more memoriesand/or databases discussed herein. Once the AI engine has thereby“learned” of an objection and/or class of objects, the AI engine may runconcurrently with and/or collaborate with other modules or applicationsdescribed herein to perform the various steps of the methods discussed.For example, in some embodiments, the AI engine recognizes an objectthat has been recognized before and stored by the AI engine. The AIengine may then communicate to another application or module of themobile device 100 an indication that the object may be the same objectpreviously recognized. IN this regard, the AI engine may provide abaseline or starting point from which to determine the nature of theobject. In other embodiments, the AI engine's recognition of an objectis accepted as the final recognition of the object.

The integrated circuit 180 may include the necessary circuitry toprovide the object recognition functionality to the mobile device 100.Generally, the integrated circuit 180 will include data storage 171which may include data associated with the objects within a real-timevideo stream that the object recognition application 125 identifies ashaving certain marker(s) (discussed in relation to FIG. 2). Theintegrated circuit 180 and/or data storage 171 may be an integratedcircuit, a microprocessor, a system-on-a-chip, a microcontroller, or thelike. As discussed above, in one embodiment, the integrated circuit 180may provide the object recognition functionality to the mobile device100.

Of note, while FIG. 1 illustrates the integrated circuit 180 as aseparate and distinct element within the mobile device 100, it will beapparent to those skilled in the art that the object recognitionfunctionality of the integrated circuit 180 may be incorporated withinother elements in the mobile device 100. For instance, the objectrecognition functionality of the integrated circuit 180 may beincorporated within the mobile device memory 120 and/or processor 110.In a particular embodiment, the object recognition functionality of theintegrated circuit 180 is incorporated in an element within the mobiledevice 100 that provides object recognition capabilities to the mobiledevice 100. Moreover, the functionality may be part of the firmware ofthe mobile device 100. Still further, the object recognitionfunctionality of the integrated circuit 180 may be included in aremovable storage device such as an SD card or the like.

The processor 110 may be configured to use the network interface 160 tocommunicate with one or more other devices on a network. In this regard,the network interface 160 may include an antenna 176 operatively coupledto a transmitter 174 and a receiver 172 (together a “transceiver”). Theprocessor 110 may be configured to provide signals to and receivesignals from the transmitter 174 and receiver 172, respectively. Thesignals may include signaling information in accordance with the airinterface standard of the applicable cellular system of the wirelesstelephone network that may be part of the network. In this regard, themobile device 100 may be configured to operate with one or more airinterface standards, communication protocols, modulation types, andaccess types. By way of illustration, the mobile device 100 may beconfigured to operate in accordance with any of a number of first,second, third, and/or fourth-generation communication protocols and/orthe like. For example, the mobile device 100 may be configured tooperate in accordance with second-generation (2G) wireless communicationprotocols IS-136 (time division multiple access (TDMA)), GSM (globalsystem for mobile communication), and/or IS-95 (code division multipleaccess (CDMA)), or with third-generation (3G) wireless communicationprotocols, such as Universal Mobile Telecommunications System (UMTS),CDMA2000, wideband CDMA (WCDMA) and/or time division-synchronous CDMA(TD-SCDMA), with fourth-generation (4G) wireless communicationprotocols, and/or the like. The mobile device 100 may also be configuredto operate in accordance with non-cellular communication mechanisms,such as via a wireless local area network (WLAN) or othercommunication/data networks.

The network interface 160 may also include an object recognitioninterface 173 in order to allow a user to execute some or all of theabove-described processes with respect to the object recognitionapplication 125 and/or the integrated circuit 180. The objectrecognition interface 173 may have access to the hardware, e.g., thetransceiver, and software previously described with respect to thenetwork interface 160. Furthermore, the object recognition interface 173may have the ability to connect to and communicate with an external datastorage on a separate system within the network as a means ofrecognizing the object(s) in the video stream.

As described above, the mobile device 100 may have a user interface thatincludes user output devices 136 and/or user input devices 140. The useroutput devices 136 may include a display 130 (e.g., a liquid crystaldisplay (LCD) or the like) and a speaker 132 or other audio device,which are operatively coupled to the processor 110. The user inputdevices 140, which may allow the mobile device 100 to receive data froma user 110, may include any of a number of devices allowing the mobiledevice 100 to receive data from a user, such as a keypad, keyboard,touch-screen, touchpad, microphone, mouse, joystick, stylus, otherpointer device, button, soft key, and/or other input device(s).

The mobile device 100 may further include a power source 115. Generally,the power source 115 is a device that supplies electrical energy to anelectrical load. In one embodiment, power source 115 may convert a formof energy such as solar energy, chemical energy, mechanical energy, etc.to electrical energy. Generally, the power source 115 in a mobile device100 may be a battery, such as a lithium battery, a nickel-metal hydridebattery, or the like, that is used for powering various circuits, e.g.,the transceiver circuit, and other devices that are used to operate themobile device 100. Alternatively, the power source 115 may be a poweradapter that can connect a power supply from a power outlet to themobile device 100. In such embodiments, a power adapter may beclassified as a power source “in” the mobile device.

The mobile device 100 may also include a memory 120 operatively coupledto the processor 110. As used herein, memory may include any computerreadable medium configured to store data, code, or other information.The memory 120 may include volatile memory, such as volatile RandomAccess Memory (RAM) including a cache area for the temporary storage ofdata. The memory 120 may also include non-volatile memory, which can beembedded and/or may be removable. The non-volatile memory mayadditionally or alternatively include an electrically erasableprogrammable read-only memory (EEPROM), flash memory or the like.

The memory 120 may store any of a number of applications or programswhich comprise computer-executable instructions/code executed by theprocessor 110 to implement the functions of the mobile device 100described herein. For example, the memory 120 may include suchapplications as an AR application 121 (described herein in relation toFIG. 3), a web browser application 122, an SMS application 123, an emailapplication 124, an object recognition application 125, etc.

Referring to FIG. 2, a block diagram illustrating an object recognitionexperience 200 in which a user 210 utilizes a mobile device 100 tocapture real-time video of an environment 250 is shown. As denotedearlier, the mobile device 100 may be any mobile communication device.The mobile device 100 has the capability of capturing real-time video ofthe surrounding environment 250. The real-time video capture may be byany means known in the art. In one particular embodiment, the mobiledevice 100 is a mobile telephone equipped with a camera 170 capable ofvideo capture.

The environment 250 contains a number of objects 220. Some of suchobjects 220 may include a marker 230 identifiable to the mobile device100. A marker 230 may be anything that is a distinguishing feature thatcan be interpreted by the mobile device 100 to identify specific objects220. For instance, a marker 230 may be alpha-numeric characters,symbols, logos, shapes, ratio of size of one feature to another feature,a product identifying code such as a bar code, electromagnetic radiationsuch as radio waves (e.g., radio frequency identification (RFID)),architectural features, color, etc. In some embodiments, the marker 230may be audio and the mobile device 100 may be capable of utilizing audiorecognition to identify words or unique sounds broadcast. The marker 230may be any size, shape, etc. Indeed, in some embodiments, the marker 230may be very small relative to the object 220 such as the alpha-numericcharacters that identify the name or model of an object 220, whereas, inother embodiments, the marker 230 is the entire object 220 such as theunique shape, size, structure, etc. of the object 220.

In some embodiments, the marker 230 is not actually a physical markerlocated on or being broadcast by the object. For instance, the marker230 may be some type of identifiable feature that is an indication thatthe object 220 is nearby. In some embodiments, the marker 230 for anobject 220 may actually be the marker 230 for a different object 220(e.g. a tag or placard near the object). In some embodiments, the marker230 for an object 220 may actually be the marker 230 for a differentobject 220. For example, the mobile device 100 may recognize aparticular building as being “Building A.” Data stored in the datastorage 171 may indicate that “Building B” is located directly to theeast and next to “Building A.” Thus, markers 230 for an object 220 thatare not located on or being broadcast by the object 220 are generallybased on fixed facts about the object 220 (e.g., “Building B” is next to“Building A”). However, it is not a requirement that such a marker 230be such a fixed fact. The marker 230 may be anything that enables themobile device 100 to interpret to a desired confidence level what theobject is. For another example, the mobile device 100, objectrecognition application 125 and/or AR application 121 may be used toidentify a particular person as a first character from a populartelevision program, and thereafter utilize the information that thefirst character is nearby features of other characters to interpret thata second character, a third character etc. are nearby, whereas withoutthe identification of the first character, the features of the secondand third characters may not have been used to identify the second andthird characters. This example may be applied to objects or businessesrather than people.

The marker 230 may also be or include social network data, such as dataretrieved or communicated from the Internet, such as micro-blog entries,blog posts, social networking site posts, various types of messagesand/or the like. In other embodiments, the marker 230 is provided inaddition to social network data as mentioned above. For example, themobile device 100 may capture a video stream and/or one or more stillshots including a large gathering of people. In this example, as above,one or more people dressed as characters in costume may be present at aspecified location. The mobile device 100, object recognitionapplication 125 and/or AR application 121 may identify several socialnetwork indicators, such as posts, blogs, micro-blog entries, messages,and/or the like indicating the presence of one or more of the specificcharacters at the specified location. In this way, the mobile device 100and associated applications may communicate information regarding thesocial media communications to the user and/or use the informationregarding the social media communications in conjunction with othermethods of object recognition. For example, the mobile device 100,object recognition application 125 and/or AR application 121 performingrecognition of the characters at the specified location may confirm thatthe object being identified is in fact the correct character based onthe retrieved social media communications. This example may also beapplied to objects or businesses rather than people.

In some embodiments, the mobile device and/or server accesses one ormore other servers, social media networks, applications and/or the likein order to retrieve and/or search for information useful in performingobject recognition. In some embodiments, the mobile device and/or serveraccesses another application by way of an application programminginterface or API. In this regard, the mobile device and/or server mayquickly search and/or retrieve information from the other programwithout requiring additional authentication steps or other gatewaysteps.

While FIG. 2 illustrates that the objects 220 with markers 230 onlyinclude a single marker 230, it will be appreciated that the object 220may have any number of markers 230 with each equally capable ofidentifying the object 230. Similarly, multiple markers 230 may beidentified by the mobile device 100 such that the combination of themarkers 230 may be utilized to identify the object 220. For example, themobile device 100 may utilize facial recognition markers 230 to identifya person and utilize a separate marker 230, such as the clothes theperson is wearing to confirm the identification to the desiredconfidence level that the person is in fact the person the mobile device100 identified. For example, the facial recognition marker 230 mayidentify the person as a famous athlete and utilize the uniform theperson is wearing to confirm that the identified person is in fact thefamous athlete.

In some embodiments, a marker 230 may be the location of the object 220.In such embodiments, the mobile device 100 may utilize GlobalPositioning System (“GPS”) hardware and/or software or some otherlocation determining mechanism to determine the location of the user 210and/or object 220. As noted above, a location-based marker 230 could beutilized in conjunction with other non-location-based markers 230identifiable and recognized by the mobile device 100 to identify theobject 230. However, in some embodiments, a location-based marker 230may be the only marker 230. For instance, in such embodiments, themobile device 100 may utilize GPS software to determine the location ofthe user 210 and a compass device or software to determine whatdirection the mobile device 100 is facing in order to identify theobject 220. In still further embodiments, the mobile device 100 does notutilize any GPS data in the identification. In such embodiments, markers230 utilized to identify the object 220 are not location-based.

FIG. 3 illustrates a mobile device 100, specifically the display 130 ofthe mobile device 100, wherein the user 210 has executed an objectrecognition application 125 and AR application 121 to present within thedisplay 130 indications of recognized objects within a real-time videostream (i.e. the surrounding environment 250). The mobile device 100 isconfigured to utilize markers 230 to identify objects 220, such asproducts that are made in the USA, the specific ingredients and/orrelative amounts of ingredients of the product, if the product is madewith recycled materials, and the like. Thereafter, the mobile device 100is configured to indicate to the user 210 identified objects 220 bydisplaying a virtual image 300 on the mobile device display 130. Asillustrated, if an object 220 does not have any markers 230 (orinsufficient markers 230 to yield object identification), the object 220will be displayed without an associated virtual image 300.

The mobile device 100 may use any type of means in order to identifydesired objects 220. For instance, the object recognition application125 may utilize one or more pattern recognition algorithms to analyzeobjects in the environment 250 and compare with markers 230 in datastorage 171 which may be contained within the mobile device 100 (such aswithin the integrated circuit 180) or externally on a separate systemaccessible via the connected network. For example, the patternrecognition algorithms may include decision trees, logistic regression,Bayes classifiers, support vector machines, kernel estimation,perceptrons, clustering algorithms, regression algorithms, categoricalsequence labeling algorithms, real-valued sequence labeling algorithms,parsing algorithms, general algorithms for predictingarbitrarily-structured labels such as Bayesian networks and Markovrandom fields, ensemble learning algorithms such as bootstrapaggregating, boosting, ensemble averaging, combinations thereof, and thelike.

Upon identifying an object 220 within the real-time video stream, themobile device 100, in some embodiments by utilizing the AR application121 is configured to superimpose a virtual image 300 on the mobiledevice display 130. The virtual image 300 is generally a tab or linkdisplayed such that the user 210 may “select” the virtual image 300 andretrieve information related to the identified object. The informationmay include any desired information associated with the selected objectand may range from basic information to greatly detailed information. Insome embodiments, the virtual image 300 may provide the user 210 with aninternet hyperlink to further information on the object 220. Theinformation may include, for example, all types of media, such as text,images, clipart, video clips, movies, or any other type of informationdesired. In yet other embodiments, the virtual image 300 informationrelated to the identified object may be visualized by the user 210without “selecting” the virtual image 300.

In embodiments in which the virtual image 300 provides an interactivetab to the user 210, the user 210 may select the virtual image 300 byany conventional means for interaction with the mobile device 100. Forinstance, in some embodiments, the user 210 may utilize an input device140 such as a keyboard to highlight and select the virtual image 300 inorder to retrieve the information. In a particular embodiment, themobile device display 130 includes a touch screen that the user mayemploy to select the virtual image 300 utilizing the user's finger, astylus, or the like.

In some embodiments, selecting the virtual image 300 will cause the ARApplication 121 to present the user 210 with the opportunity to purchasethe object associated with the virtual image. In some instances, theorder functionality may be a part of the AR Application 121. In otherembodiments, the AR Application may access the user's accountsassociated with a financial institution enabling the user 210 topurchase the object 220 without having to input data such as a creditcard number of billing address. In other embodiments, when the user 210selects the virtual image 300 to purchase the product, the ARApplication 121 will activate the web browser application 122 to allowthe user to purchase the product over the Internet from a websiteassociated with the object 220.

In some embodiments, the virtual image 300 is not interactive and simplyprovides information to the user 210 by superimposing the virtual image300 onto the display 130. For example, in some instances it may bebeneficial for the AR application 121 to merely identify an object 220,providing the object's name/title and giving brief information about theobject, etc., rather than provide extensive detail that requiresinteraction with the virtual image 300. The mobile device 100 is capableof being tailored to a user's desired preferences.

Furthermore, the virtual image 300 may be displayed at any size on themobile device display 130. The virtual image 300 may be small enoughthat it is positioned on or next to the object 220 being identified suchthat the object 220 remains discernable behind the virtual image 220.Additionally, the virtual image 300 may be semi-transparent such thatthe object 220 remains discernable behind the virtual image. In otherembodiments, the virtual image 220 may be large enough to completelycover the object 220 portrayed on the display 130. Indeed, in someembodiments, the virtual image 220 may cover a majority or the entiretyof the mobile device display 130.

The user 210 may opt to execute the object recognition application 125and AR application 121 at any desired moment and begin video capture andanalysis. However, in some embodiments, the object recognitionapplication 125 and the AR application 121 includes an “always on”feature in which the mobile device 100 is continuously capturing videoand analyzing the objects 220 within the video stream. In suchembodiments, the object recognition application 125 may be configured toalert the user 210 that a particular object 220 has been identified. Theuser 210 may set any number of user preferences to tailor the objectrecognition and AR experience to their needs. For instance, the user 220may opt to only be alerted if a certain particular object 220 isidentified. The user 220 may also opt to have the video captured upon aspecific manipulation of the system, hand gesture, voice commend, etc.by the user 220. Additionally, it will be appreciated that the “alwayson” feature in which video is continuously captured may consume themobile device power source 115 more quickly. Thus, in some embodiments,the “always on” feature may disengage if a determined event occurs suchas low power source 115, low levels of light for an extended period oftime (e.g., such as if the mobile device 100 is in a user's pocketobstructing a clear view of the environment 250 from the mobile device100), if the mobile device 100 remains stationary (thus receiving thesame video stream) for an extended period of time, the user sets acertain time of day to disengage, etc. Conversely, if the “always on”feature is disengaged due to the occurrence of such an event, the user210 may opt for the “always on” feature to re-engage after the durationof the disengaging event (e.g., power source 115 is re-charged, lightlevels are increased, etc.).

In some embodiments, the user 210 may identify objects 220 that theobject recognition application 125 does not identify and add it to thedata storage 171 with desired information in order to be identifiedand/or displayed in the future. For instance, the user 210 may select anunidentified object 220 and enter a name/title and/or any other desiredinformation for the unidentified object 220. In such embodiments, theobject recognition application 125 may detect/record certain markers 230about the object 220 so that the pattern recognition algorithm(s) (orother identification means) may detect the object 220 in the future.Furthermore, in cases where the object information is within the datastorage 171, but the object recognition application 125 fails toidentify the object 220 (e.g., one or more identifying characteristicsor markers 230 of the object has changed since it was added to the datastorage 171 or the marker 230 simply was not identified), the user 210may select the object 220 and associate it with an object 220 alreadystored in the data storage 171. In such cases, the object recognitionapplication 125 may be capable of updating the markers 230 for theobject 220 in order to identify the object in future real-time videostreams.

In addition, in some embodiments, the user 210 may opt to edit theinformation or add to the information provided by the virtual object300. For instance, the user 210 may opt to include user-specificinformation about a certain object 220 such that the information may bedisplayed upon a future identification of the object 220. Conversely, insome embodiments, the user may opt to delete or hide an object 220 frombeing identified and a virtual object 300 associated therewith beingdisplayed on the mobile device display 130.

Furthermore, in some instances, an object 220 may include one or moremarkers 230 identified by the object recognition application 125 thatleads the object recognition application 125 to associate an object withmore than one object in the data storage 171. In such instances, theuser 210 may be presented with the multiple candidate identificationsand may opt to choose the appropriate identification or input adifferent identification. The multiple candidates may be presented tothe user 210 by any means. For instance, in one embodiment, thecandidates are presented to the user 210 as a list wherein the“strongest” candidate is listed first based on reliability of theidentification. Upon input by the user 210 identifying the object 220,the object recognition application 125 may “learn” from the input andstore additional markers 230 in order to avoid multiple identificationcandidates for the same object 220 in future identifications.

Additionally, the object recognition application 125 may utilizecriteria for identification other than identification algorithms. Forinstance, the object recognition application 125 may utilize the user'slocation, time of day, season, weather, speed of location changes (e.g.,walking versus traveling), “busyness” (e.g., how many objects are inmotion versus stationary in the video stream), social networkingupdates, as well any number of other conceivable factors in determiningthe identification of objects 220. Moreover, the user 210 may inputpreferences or other metrics for which the object recognitionapplication 125 may utilize to narrow results of identified objects 220.

In some embodiments, the AR application 121 may have the ability togather and report user interactions with displayed virtual objects 300.The data elements gathered and reported may include, but are not limitedto, number of offer impressions; time spent “viewing” an offer, product,object or business; number of offers investigated via a selection;number of offers loaded to an electronic wallet and the like. Such userinteractions may be reported to any type of entity desired. In oneparticular embodiment, the user interactions may be reported to afinancial institution and the information reported may include customerfinancial behavior, purchase power/transaction history, and the like.

In various embodiments, information associated with or related to one ormore objects that is retrieved for presentation to a user via the mobiledevice may be permanently or semi-permanently associated with theobject. In other words, the object may be “tagged” with the information.In some embodiments, a location pointer is associated with an objectafter information is retrieved regarding the object. In this regard,subsequent mobile devices capturing the object for recognition mayretrieve the associated information, tags and/or pointers in order tomore quickly retrieve information regarding the object. In someembodiments, the mobile device provides the user an opportunity to postmessages, links to information or the like and associate such postingswith the object. Subsequent users may then be presenting such postingswhen their mobile devices capture and recognize an object. In someembodiments, the information gathered through the recognition andinformation retrieval process may be posted by the user in associationwith the object. Such tags and/or postings may be stored in apredetermined memory and/or database for ease of searching andretrieval.

FIG. 5 illustrates a process flow 500 for determining which objects 220(FIGS. 2 and 3) from the real-time video stream are associated withgoods or business consistent with a predetermined social impact profile,in accordance with an embodiment of the invention. As represented byblock 510, data is collected concerning the social values of greatestimportance to the user and this data is used to generate a social impactprofile for the user, as represented by block 520. Block 530 representsthe identification of goods and businesses captured in the real-timevideo stream consistent with the systems and apparatus discussed aboveand illustrated in FIGS. 1-3.

As represented by block 540 the goods and businesses identified in thereal-time video stream are correlated to information regarding thesocial impact of the goods and businesses. This information is used toassign the identified goods and businesses a social impact profile, asrepresented in block 550. In some embodiments of the invention, theinformation regarding the social impact of the goods and businesses maybe dynamically collected by using search algorithms to pair the markers230 to information in existing data sources related to the social impactinformation about the goods or businesses. Such data sources mayinclude, but are not limited to the Internet, legal records, newsdatabases, and the like. In some embodiments a database will bespecifically created to catalog the social impact information of goodsand businesses. In some embodiments this social impact database may bestored on the Data Storage 171 of the mobile device 100 to be accessedby the processor 110 as part of the object recognition application 125.In some embodiments, the social impact database may be provided to thesystem through an API. In this way, the data may be stored in a separateAPI and be implemented by request from the mobile device and/or serveraccesses another application by way of an API. In some embodiments, afinancial institution may use its unique relationship with bothindividuals and businesses to collect social impact information of goodsand businesses for use in connection with apparatuses and systemsperforming the process flow 500. It will be understood that althoughprocess flow 500 illustrates the correlation of goods and businesses toinformation regarding the social impact of the goods and businesses 540as a separate step from assigning a social impact profile to the goodsand businesses 550 that these steps may be performed simultaneously.Moreover, one or both of these steps may be performed prior to the othersteps shown in the process flow 500. In some embodiments, informationregarding the social impact of goods and businesses will be collectedand the goods and businesses will be assigned a social impact profilethat is preloaded onto the mobile device 100.

Referring back to the process flow 500 illustrated in FIG. 5, block 560represents the event of comparing the social impact profile of the goodsor businesses to the social impact profile of the user. As discussedabove, in some embodiments, the social impact profile may take the formof a numerical indicator. Accordingly, in some embodiments the numericalindicator assigned to the user will be compared to the numericalindicator assigned to the goods or business. In some embodiments, avirtual image 300 will only be shown on the display 130 if the object220 has a social impact profile greater than the social impact profileof the user. For example, if a user is assigned a social impact profileof forty-two on a scale of one hundred for environmental concerns and anidentified product has a social impact profile of fifty-five a virtualimage 300 for the product will be shown on the display 130 of the mobiledevice 100. If however, the product has score of thirty-five, no virtualimage 300 will be shown on the display 100. It will be understood, thatif the social impact profile takes an alternate form, such as aclassification or grade, the comparison will proceed similarlyidentifying goods and businesses with a social impact profile that is ata specific level relative to the user's social impact profile. The userwill be able to adjust and specify the relationship between the socialimpact profiles that will still generate a virtual image 300. Forinstance, a user may be able to specify that it will accept goods orbusinesses with social impact profile within a certain percentage of itsown, or if classifications are used, that the user will accept goodsfrom the classification immediately below the user's. In otherembodiments, a virtual image 300 will be outlined, for example in green,if the object 220 in the display has a social impact profile appropriateto the user's assignment.

The collection of data concerning the user's social values representedin block 510 of the process flow 500 can be embodied in a number ofdifferent apparatuses and systems. FIG. 6 provides a flow diagram 600illustrating a general process flow of an apparatus or system forcollecting data concerning the social values of greatest importance tothe user 510. In some embodiments, an apparatus, such as the mobiledevice 100 will be used to collect user defined data regarding thesocial values of importance to the user 610. Such user defined data maytake a number of different forms, including but not limited to,responses to survey questions and questionnaires regarding socialvalues, self-identification with specific social groups, identificationof search terms that a user wants, or does not want to be associatedwith products or businesses, etc. In some embodiments, such user defineddata may be collected directly by the mobile device 100 as part of theobject recognition application 125 or through a website accessed throughthe web browser application 122. In some embodiments the user defineddata is collected through other computer systems, or even through directinterviews with the user and are subsequently transferred to the datastorage 171.

As represented by block 620 the apparatus or system may analyze thepurchasing behavior of the user to identify the social values ofimportance to the user. Past purchasing behavior may be a good indicatorof the social values of greatest importance to a user, or at a minimumreflect the social impact values on which a user is prepared to act.Although information regarding purchasing behavior may be collected fromdifferent sources, in some embodiments, a financial institution willutilize its relationship with its customers to collect and analyze theuser's purchasing behavior. If a user has an account with a financialinstitution, the financial institution will have access to transactioninformation for purchases made by the user from that account. Suchtransaction information may include the date, time, location, amount,nature, product details (i.e. merchant category code), and provider ofproducts or services purchased. This transaction information can then becompared to information relating to the social impact of specificproducts and businesses. In some embodiments, this information will bethe same information as represented in block 540 for process flow 500 inFIG. 5 and discussed above. Accordingly, a database may be created bythe financial institution or another entity cataloging the social impactof various goods and businesses or the information may be collecteddynamically from existing sources. The financial institution can thencompare the user's transaction data to the available social impactinformation. For example, if a high percentage of a user's purchases,either with regard to dollar amount or frequency, are from a grocerystore that is identified as having a social impact profile associatedwith selling organic goods, the apparatus or system may categorize theuser as valuing organic products. If a user has used an account to makedonations to a non-profit organization, the apparatus may categorize theuser as valuing charitable giving or may associate the user with thespecific social values identified with the non-profit organization thatreceived the donation.

In some embodiments of the invention, the use of purchasing behavior tocollect information regarding the social values of greatest importanceto the user will also allow the user to gain information regarding theuser's own activities or habits. In some embodiments, the objectrecognition application 125 or some other application on the memory 120of the mobile device 100 or other computing system can provide a summaryof the analysis of the user's purchasing behavior. For instance, theobject recognition application 125 may present the user's checkingaccount statement or credit card statement and highlight purchases thatreflect a specific social impact profile or are inconsistent with aspecific social impact profile. The object recognition application 125or other program may also generate a social impact score. This score mayor may not be the same as the social impact profile represented by block520. The object recognition application 125 or other program mayidentify those purchases that contributed to the specific score so thatthe user can adjust his or her behavior to move the score in a directionconsistent with the user's self-defined values. In some embodiments ofthe invention, the user's social impact score will be sent to socialnetworking accounts associated with the user to enable to the user toshare his or her score with friends and associates. Some embodiments ofthe invention will include a game feature providing incentives forimproving a social impact score.

Referring back to FIG. 6, block 630 represents the process flow ofanalyzing the user's personal profile to identify the social values ofimportance to the user. As used herein “personal profile” means anyinformation personal to the user that can be used to identify thecharacter and quality of the person. Such information includes but isnot limited to, membership in certain organizations, profile informationavailable through any social network of the user, status updates, blogpostings, comments and other information contributed by the user in anonline environment, contents of e-mails etc. In some embodiments of theinvention, a computer device processor will be used to monitor or browsethrough all electronic information related to the user and collectpersonal profile information associated with the user's social values.In some embodiments, a financial institution will use the information ithas access to by nature of its relationship with its customers tocollect a user's personal profile information. Some personal profileinformation is provided directly to the financial institution throughthe process of establishing accounts and doing business with thefinancial institution. The financial institution may also have access tocertain transaction information that indicates personal profileinformation related to the user's social values, such as the payment ofmembership dues to certain organizations or the contribution of funds toa non-profit organization.

As with the analysis of purchasing behavior represented in block 620,the analysis of personal profile information represented in block 630will allow the user, in certain embodiments to gain information aboutthe user's social values. In certain embodiments, the object recognitionapplication 125 or other program on the mobile device 100 or othercomputing system will present to the user the effect certain personalprofile information has on the determination of the user's socialvalues. For instance, the object recognition application 125 mayhighlight information from a user's blog post or social networkingprofile that indicates the user values local businesses.

Referring now to FIGS. 7-9, a series of exemplary display pages 700 and800 of a mobile device 100 consistent with an embodiment of the presentinvention are provided. It will be understood, that in some embodiments,each of the pages 700 and 800 are displayed on the display 130 of themobile device 100. It will also be understood that the mobile device 100is configured, in some embodiments to navigate from page 700 to page 800and/or vice versa either by actuating one of the buttons 730 or one ofthe other input devices 140. It will be further understood that thedisplay pages 700 and 800 can be embodied as portions of the objectrecognition application 125, the AR Application 121, as Internet webpages and/or the like. In addition, it will be understood that, in someembodiments, the apparatus having the process flow 400, 500 and 600 areconfigured to implement any one or more of the embodiments of thepresent invention described and/or contemplated herein in connectionwith the display pages 700 and 800.

Referring now to FIG. 7, it will be understood that in some embodiments,the display page 700 is displayed in connection with collecting userdefined data regarding the social values of importance to the user, asrepresented by block 610 of the apparatus having a process flow 600.Alternatively, the display page 700 is displayed after the customernavigates to the page from the other display page 800. As shown, in thisembodiment, the display page 700 includes a general survey question andinstructions 705, specific categories for inputting user definedresponses to social impact queries 710 and response indicators 720. Inthe embodiment shown in display page 700, the user is asked to identifyhow important certain factors are to making a purchasing decision withone being very important and five being not important at all 705. Thespecific factors are presented at 710 and the user indicates his or herresponse by actuating one response indicator 720 for each category. Oncethe user has provided a response for each category, the application orprogram will provide an additional question to solicit informationregarding the social values of the customer or the customer will be ableto navigate to one of other display pages, such as display page 800.

Referring now to FIG. 8, it will be understood that in some embodiments,the display page 800 is displayed in connection with collecting userdefined data regarding the social values of importance to the user asrepresented by block 610 of the apparatus having a process flow 600.Alternatively, the display page 800 is displayed after the usernavigates to the page from the other display page 700. As shown, in thisembodiment, the display page 800 includes instructions 810 a field forinputting search terms 720 and a button 830 for submitting the searchterms for use in connection with identification of products orbusinesses having a predetermined social impact as discussed herein. Inthis embodiment, the user may input any terms that the user either wantsto be associated with specific products or businesses or does not wantto be associated with specific products or businesses. The user has theoption of selecting whether the AR Application 121 returns virtualimages 300 for goods positively or negatively associated with the searchterms (not shown). For instance, the user may enter the terms“environmentally friendly” or “locally grown” in the search field 820and select the input for positive association and the object recognitionapplication 125 will identify goods in the real-time video stream thathave been identified as being environmentally friendly or locally grownand the AR Application 121 may superimpose a virtual image 300 on thedisplay over the corresponding image of such goods. The user may alsoenter “blood diamond” in the search field 820 and select the input fornegative association and the object recognition application 125 willexclude any goods or businesses that have been associated with diamondsremoved from certain conflict zones or mined in certain geographiclocations. It will be understood that the search terms can be combinedusing conventional Boolean operators and the like.

Referring now to FIG. 9, which is an exemplary display page 900 of amobile device 100 consistent with an embodiment of the presentinvention. It will be understood, that in some embodiments, the displaypage 900 is displayed on the display 130 of the mobile device 100. Itwill also be understood that the display pages 900 can be embodied asportions of the object recognition application 125, the AR Application121, as Internet web pages and/or the like. In addition, it will beunderstood that, in some embodiments, the apparatus having the processflow 400, 500 and 600 are configured to implement any one or more of theembodiments of the present invention described and/or contemplatedherein in connection with the display pages 900.

The display page 900 of FIG. 9, in some embodiments is displayed inconnection with an apparatus performing block 430 of the process flow400. Referring to FIG. 3, after identifying an object 220 within thereal-time video stream, the mobile device 100 is configured tosuperimpose a virtual image 300 on the mobile device display 130. Insome embodiments selecting the virtual image 300 navigates the user todisplay page 900 to enable the user to purchase the identified productor service. As shown, in this embodiment, the display page 900 includesan image of the product or service 910 a description of the product orservice 920 the price of the product or service 930 and a button toinitiate the purchase of the product or service 940. The display page900 may also include additional information 950 regarding the product orservice, including such things as reviews, specifications, instructionsand multimedia relating to the product or service. In certainembodiments of the invention, selecting the button 940 will navigate theuser to an additional display page (not shown) where the user can enterinformation, such as credit card numbers and billing addresses, tocomplete the purchase. In other embodiments, the AR Application will belinked to the financial institution of the user so by actuating thebutton 940 the purchase is automatically completed and no additionalinformation is required from the user.

While the foregoing disclosure discusses illustrative embodiments, itshould be noted that various changes and modifications could be madeherein without departing from the scope of the described aspects and/orembodiments as defined by the appended claims. Furthermore, althoughelements of the described aspects and/or embodiments may be described orclaimed in the singular, the plural is contemplated unless limitation tothe singular is explicitly stated. Additionally, all or a portion of anyembodiment may be utilized with all or a portion of any otherembodiment, unless stated otherwise.

The systems, methods, computer program products, etc. described herein,may be utilized or combined with any other suitable AR-relatedapplication. Non-limiting examples of other suitable AR-relatedapplications include those described in the following U.S. ProvisionalPatent Applications, the entirety of each of which is incorporatedherein by reference:

U.S. Provisional Ser. No. Filed On Title 61/450,213 Mar. 8, 2011Real-Time Video Image Analysis Applications for Commerce Activity61/478,409 Apr. 22, 2011 Presenting Offers on a Mobile CommunicationDevice 61/478,412 Apr. 22, 2011 Real-Time Video Analysis for RewardOffers 61/478,394 Apr. 22, 2011 Real-Time Video Image Analysis forProviding Targeted Offers 61/478,399 Apr. 22, 2011 Real-Time AnalysisInvolving Real Estate Listings 61/478,402 Apr. 22, 2011 Real-Time VideoImage Analysis for an Appropriate Payment Account 61/478,405 Apr. 22,2011 Presenting Investment-Related Information on a Mobile CommunicationDevice 61/478,393 Apr. 22, 2011 Real-Time Image Analysis for MedicalSavings Plans 61/478,397 Apr. 22, 2011 Providing Data Associated WithRelationships Between Individuals and Images 61/478,408 Apr. 22, 2011Identifying Predetermined Objects in a Video Stream Captured by a MobileDevice 61/478,400 Apr. 22, 2011 Real-Time Image Analysis for ProvidingHealth Related Information 61/478,411 Apr. 22, 2011 Retrieving ProductInformation From Embedded Sensors Via Mobile Device Video Analysis61/478,407 Apr. 22, 2011 Providing Information Associated With anIdentified Representation of an Object 61/478,415 Apr. 22, 2011Providing Location Identification of Associated Individuals Based onIdentifying the Individuals in Conjunction With a Live Video Stream61/478,419 Apr. 22, 2011 Vehicle Recognition 61/478,417 Apr. 22, 2011Collective Network of Augmented Reality Users 61/508,985 Jul. 18, 2011Providing Information Regarding Medical Conditions 61/508,946 Jul. 18,2011 Dynamically Identifying Individuals From a Captured Image61/508,980 Jul. 18, 2011 Providing Affinity Program Information61/508,821 Jul. 18, 2011 Providing Information Regarding SportsMovements 61/508,850 Jul. 18, 2011 Assessing EnvironmentalCharacteristics in a Video Stream Captured by a Mobile Device 61/508,966Jul. 18, 2011 Real-Time Video Image Analysis for Providing VirtualLandscaping 61/508,969 Jul. 18, 2011 Real-Time Video Image Analysis forProviding Virtual Interior Design 61/508,971 Jul. 18, 2011 Real-TimeVideo Image Analysis for Providing Deepening Customer Value 61/508,764Jul. 18, 2011 Conducting Financial Transactions Based on Identificationof Individuals in an Augmented Reality Environment 61/508,973 Jul. 18,2011 Real-Time Video Image Analysis for Providing Security 61/508,976Jul. 18, 2011 Providing Retail Shopping Assistance 61/508,944 Jul. 18,2011 Recognizing Financial Document Images

While certain exemplary embodiments have been described and shown in theaccompanying drawings, it is to be understood that such embodiments aremerely illustrative of and not restrictive on the broad invention, andthat this invention not be limited to the specific constructions andarrangements shown and described, since various other changes,combinations, omissions, modifications and substitutions, in addition tothose set forth in the above paragraphs, are possible. Those skilled inthe art will appreciate that various adaptations and modifications ofthe just described embodiments can be configured without departing fromthe scope and spirit of the invention. Therefore, it is to be understoodthat, within the scope of the appended claims, the invention may bepracticed other than as specifically described herein.

1. A method of providing social impact information of goods and servicesproviders, the method comprising: identifying, via a processor, one ormore goods or service providers proximate in location to a mobiledevice; determining, via the processor, which goods or service providersproximate in location to the mobile device are consistent with apredetermined social impact; and presenting, via a display, one or moreindicators, each indicator associated with a good or service providerproximate in location to the mobile device determined to be consistentwith the predetermined social impact.
 2. The method of claim 1, whereinidentifying one or more goods or service providers further comprisescapturing, via the mobile device, a real-time video stream, wherein thereal-time video stream includes at least one image of a good or serviceprovider.
 3. The method of claim 2, wherein capturing a real-time videostream further comprises implementing, via the processor, objectrecognition processing to identify the goods or service providers withinthe real-time video stream.
 4. The method of claim 1, whereinidentifying one or more goods or service providers further comprisescapturing a code located on or proximate to one or more of the productsor service providers, wherein the code identifies the product or serviceprovider.
 5. The method of claim 1, wherein identifying one or moregoods or service providers further comprises receiving, via the mobiledevice, wireless communication from a product identifying tag or aservice provider identifying tag associated with the goods or serviceprovider.
 6. The method of claim 1, wherein identifying one or moregoods or service providers further comprises determining, via theprocessor, a location of the mobile device and determining, via theprocessor, the one or more goods or service providers based on thedetermined location.
 7. The method of claim 1, wherein the mobile deviceis a mobile phone.
 8. The method of claim 2, wherein the real-time videostream is captured by the same mobile device that presents the one ormore indicators.
 9. The method of claim 1, wherein determining whichgoods or service providers proximate in location to the mobile deviceare consistent with a predetermined social impact further comprises:collecting data concerning the social values of importance to a user;and generating a social impact profile for the user.
 10. The method ofclaim 2, wherein determining which images from the real-time videostream are associated with goods or service providers consistent with apredetermined social impact further comprises: identifying the goods andservice providers captured in the real-time video stream; correlatingthe goods and service providers identified in the video stream toinformation regarding the social impact of the goods or serviceproviders; comparing the information regarding the social impact of thegoods and service providers to the social impact profile of the user.11. The method of claim 9 wherein collecting the data concerning thesocial values of importance to the user comprises collecting userdefined data concerning the social values of importance to the user 12.The method of claim 9 wherein collecting the data concerning the socialvalues of importance to the user comprises: analyzing the purchasingbehavior of the user; and identifying the social values of importance tothe user based on the purchasing behavior of the user.
 13. The method ofclaim 9 wherein collecting the data concerning the social values ofimportance to the user comprises: collecting personal data associatedwith the user; and identifying the social values of importance to theuser based on the personal data of the user.
 14. The method of claim 10wherein the social impact of the goods or service providers comprisesone or more of the following: an environmental impact, nature ofproduction, product attributes, business attributes or charitableactivities.
 15. The method of claim 11 wherein the user defined datacomprises responses to survey questions.
 16. The method of claim 11wherein the user defined data comprises search terms inputted by theuser and the user's desired correlation between the inputted searchterms and the goods or service providers.
 17. The method of claim 12wherein the purchasing behavior comprises financial records available toa financial institution.
 18. The method of claim 13 wherein the personaldata associated with the user comprises data available to a financialinstitution.
 19. The method of claim 2 wherein presenting, via adisplay, one or more indicators, each indicator associated with a goodor service provider proximate in location to the mobile devicedetermined to be consistent with the predetermined social impactcomprises superimposing the indicator over the real-time video stream.20. The method of claim 1, wherein the indicators are selectable by auser.
 21. The method of claim 1, wherein the indicators, upon beingselected present a user with additional information about the goods orservice provider.
 22. The method of claim 1, wherein the indicators,upon being selected present a user with an option to purchase one ormore of the goods or a service provided by the service provider via themobile device.
 23. An apparatus comprising: a device for capturing areal-time video stream, wherein the real-time video stream includes atleast one image of a good or service provider; a processor operablyconnected to the device, configured to determine which images from thereal-time video stream are associated with goods or service providersconsistent with a predetermined social impact profile; and a display,operably connected to the processor, for presenting the real-time videostream with one or more indicators, each indicator associated with animage of a good or service provider determined to be consistent with thepredetermined social impact profile.
 24. The apparatus of claim 19,wherein the device is a mobile device.
 25. The apparatus of claim 19,wherein the device is a mobile phone.
 26. An apparatus as in claim 19,wherein the processor, in determining which images from the real-timevideo stream are associated with goods or service providers consistentwith a predetermined social impact profile, is further configured to:collect data concerning the social values of importance to a user;generate a social impact profile for the user; identify goods andservice providers captured in the real-time video stream; correlate thegoods or service providers captured in the real-time video stream toinformation regarding the social impact of the goods or serviceproviders; assign a social impact profile to the goods or serviceprovider captured in the real-time video stream; and correlate thesocial impact profile for the user to the social impact profile of thegoods or service provider.
 27. The apparatus of claim 22 wherein theprocessor in collecting data concerning the social values of importanceto the user is configured to collect user defined data concerning thesocial values of importance to the user.
 28. The apparatus of claim 22wherein the processor in collecting data concerning the social values ofimportance to the user is configured to: analyze the purchasing behaviorof the user; and identify the social values of importance to the userbased on the purchasing behavior of the user.
 29. The apparatus of claim22 wherein the processor in collecting data concerning the social valuesof importance to the user is configured to: collect personal dataassociated with the user; and identify the social values of importanceto the user based on the purchasing behavior of the user.
 30. Theapparatus of claim 22 wherein the social impact of the goods or serviceprovider comprise one or more of the following: environmental impact,nature of production, product attributes, business attributes orcharitable activities
 31. The apparatus of claim 23 wherein the userdefined data comprises responses to survey questions.
 32. The apparatusof claim 23 wherein the user defined data comprises search termsinputted by the user and the user's desired correlation between theinputted search terms and the goods or service providers.
 33. Theapparatus of claim 24 wherein the purchasing behavior comprisesfinancial records available to a financial institution.
 34. Theapparatus of claim 24 wherein the personal data associated with the usercomprises data available to a financial institution.
 35. The apparatusof claim 19 wherein the display in presenting the real-time video streamwith one or more indicators is further configured to superimpose theindicator over the image of the goods or service providers determined tobe consistent with the predetermined social impact profile.
 36. Themethod of claim 19, wherein the indicators are selectable by the user.37. The method of claim 19, wherein the indicators, upon begin selectedpresent the user with additional information about the goods or serviceproviders.
 38. The method of claim 19, wherein the indicators, uponbeing selected present the user with an option to purchase one or moreof the goods or a service provided by the service provider via thedevice.
 39. A computer program product, comprising: a non-transitorycomputer-readable medium having computer-executable code stored thereon,the computer executable code comprising: a first code portion configuredto capture, via a mobile device, a real-time video stream, a second codeportion configured to determine which images from the real-time videostream are associated with goods or service providers consistent with apredetermined social impact profile, a third code portion configured topresent on a display the real-time video stream with one or moreindicators, each indicator associated with an image determined to beconsistent with the predetermined social impact profile.
 40. Thecomputer program product of claim 35, wherein the second code portion isfurther configured to: collect data concerning the social values ofimportance to the user; generate a social impact profile for the user;identify the goods and service providers captured in the real-time videostream; correlate the goods and service providers identified in thevideo stream to information regarding the social impact of the goods andbusinesses; comparing information regarding the social impact of thegoods and businesses to the social impact profile of the user.
 41. Thecomputer program product of claim 36, wherein collecting the dataconcerning the social values of importance to the user comprisescollecting user defined data concerning the social values of importanceto the user.
 42. The computer program product of claim 36, whereincollecting the data concerning the social values of importance to theuser comprises: analyzing the purchasing behavior of the user; andidentifying the social values of importance to the user based on thepurchasing behavior of the user.
 43. The computer program product ofclaim 36 wherein collecting the data concerning the social values ofgreatest importance to the user comprises: collecting personal dataassociated with the user; and identifying the social values ofimportance to the user based on the purchasing behavior of the user. 44.The computer program product of claim 36 wherein the social impact ofthe goods or service providers comprise one or more of the following:environmental impact, nature of production, product attributes, businessattributes or charitable activities.
 45. The computer program product ofclaim 37, wherein the user defined data comprises responses to surveyquestions.
 46. The computer program product of claim 37, wherein theuser defined data comprises search terms inputted by the user and theuser's desired correlation between the inputted search terms and thegoods or service providers.
 47. The computer program product of claim38, wherein the purchasing behavior comprises financial recordsavailable to a financial institution.
 48. The computer program productof claim 39, wherein the personal data associated with the usercomprises data available to a financial institution.
 49. The computerprogram product of claim 35, wherein third code portion is configured tosuperimpose the indicators over the real-time video stream.
 50. Thecomputer program product of claim 35, wherein the indicators areselectable by the user.
 51. The computer program product of claim 35,wherein the indicators, upon being selected present the user withadditional information about the goods or service providers.
 52. Thecomputer program product of claim 35, wherein the indicators, upon beingselected present the user with an option to purchase one or more of theproducts or services provided by the service provider via the mobiledevice.
 53. The computer program product of claim 35, wherein the mobiledevice is a mobile phone.